test: an involuntary skip is a failure — tools/_skip_policy.py and the rig lane (#178a) - #180

Add tools/_skip_policy.py and wire the in-scope entry points to it.  Three
lanes, deliberately different calls, because the REMEDY decides the verdict:

  not_applicable() VOLUNTARY -- this host or configuration can never run
                  this rig and another rig owns the coverage, or the
                  operator declined an opt-in rig.  Nothing is lost, so
                  exit 0 -- but as a named verdict, never a bare
                  `return 0`.  Remedy: "go use the other rig", or "opt in".
  cannot_run()    INVOLUNTARY -- this IS the target platform/config and it
                  still could not run: no toolchain, no PRG, no hardware.
                  A real coverage hole, so exit 2 (distinct from 1, "a
                  check ran and failed").  Remedy: "install something".
  cannot_run(opt_out_env=None) -- the two things no environment variable
                  may silence: a failed build, and contention.

Measured before the change, with build/ absent:

  tests/rig_phase{1_dhcp,2_http,3_https,3_https_1mhz}.py   exit 0 at the
    missing-prerequisite site AND at `return _skip("c64-https.prg could
    not be built")` -- a failed `make` laundered into a pass.
  tests/rig_vice_https_macos.py                            exit 0 with the
    rig down; and `SystemExit("build failed")` exited 1, making a broken
    build indistinguishable from a real handshake failure.
  tools/test_p384_symbols.py                               exit 0 on
    BACKEND != uci (voluntary, but silent).
  tools/test_ecdsa_p384_kat.py --u64 with no U64_HOST       printed
    "U64: 0/0 passed" and OVERALL: PASS, exit 0.

BEHAVIOUR CHANGE worth stating rather than discovering: that last one is
now refused UP FRONT, before the multi-hour VICE lane, instead of at the
verdict.  `--u64` without U64_HOST fails in ~0.1 s.

Platform is asked FIRST, and it is a voluntary skip.  The four bridge rigs
are Linux-only; on Darwin check_prerequisites() reports "ip not on PATH;
iptables not on PATH", which is not installable there, so routing that to
cannot_run() would make four rigs a permanent red on the project's primary
platform -- with the global opt-out as the only remedy, i.e. the policy
gets disabled everywhere.  https_e2e gains a separate platform_supported()
predicate rather than partitioning check_prerequisites(): that function
returns a flat list of strings whose entries mean different things on
different platforms, and it is shared by four rigs and shadowed by a
same-named function in tools/test_http_integration.py.  Its contract is
unchanged.  rig_vice_https_macos.py gets the symmetric lift.

Contention is a third category.  In rig_vice_https_macos.py it is split
out of _rig_check(); in rig_phase3_https_1mhz.py the port-443 check is the
same thing.  Both exit 2 with no opt-out: a lane that silences contention
goes green exactly when it collides, and unlike a missing tool it clears
on its own.  It is evaluated AFTER `problems`, because contention on a rig
that does not exist is a meaningless headline -- and, being non-opt-out-
able, would otherwise leave a no-rig CI lane red with no recourse.

The contention detector is narrowed accordingly.  `pgrep -fl "ethernetioif
feth0"` matched the full command line of every process against an
unanchored pattern, so a grep, an editor, or a driver script carrying that
text matched too (demonstrated: a plain `python3 -c ... "ethernetioif
feth0"` matches).  It is now `pgrep -x x64sc` plus an argv check, so it
matches only a process whose executable is x64sc and whose arguments name
feth0.

rig_phase3_https_1mhz.py's two pre-flight gates predate this change but
contradicted the contract it documents: an unset VICE_HTTPS_OK_TO_RUN is
the operator declining an opt-in rig (now exit 0, named verdict), and a
held port 443 is contention (exit 2, no opt-out).  The platform gate moved
ahead of both, so a host that can never run the rig is never told to opt
in and never trips the port check.

The opt-out requires the literal "1".  Bare truthiness meant
C64_ALLOW_SKIP=0 ENABLED it, so setting 0 to close the hatch disabled the
whole policy; every other gate in this repo (C64_SKIP_BUILD,
VICE_HTTPS_OK_TO_RUN, ...) compares to "1" and this one now does too.
require() also fails closed when the opt-out is set but pytest is absent,
rather than returning into a test body whose prerequisite is missing.

Two test modules, both pinned in pytest.ini testpaths:

  tools/test_skip_policy.py       15 cases -- the "1" comparison across
    0 / false / empty / yes / true / 2 / " 1 " / "1\n", both lanes, and
    require().
  tools/test_rig_skip_contract.py 10 cases -- the CALL SITES.  A helper
    with no call-site test is just another convention to miss, which is
    #178's own thesis: nothing otherwise pinned that a rig asks
    platform_supported() BEFORE check_prerequisites(), and swapping those
    two lines silently returns macOS to exit 2 with a green suite.  Both
    ordering assertions were mutation-tested: swapping the gates in
    rig_phase1_dhcp.py trips the tripwire naming check_prerequisites, and
    swapping problems/contention in rig_vice_https_macos.py fails the
    routing assertion.

No `total_run == 0` vacuity guard is left in test_ecdsa_p384_kat.py.  One
was written and removed: _build_vector_list() is never empty and
_run_backend() counts every vector, including under --sha-only, so it
could not fire under any flag combination.  A check that matches nothing
is the shape #178 exists to close, and shipping one inside #178's own
implementation is the worst place for it.

tools/test_rig_skip_contract.py does NOT import tools/https_e2e at module
level.  That package re-exports from .vice_on_bridge, which imports
c64_test_harness from a sibling checkout, so a module-level import made bare
`pytest` on a fresh clone die with a COLLECTION ERROR -- which pytest.ini's
own comment says must never be mistaken for a passing run -- and it silently
voided the very hazard the module's docstring cites as its reason not to
import the macOS rig.  It is now loaded on first use through require(), the
API's first production call site: with c64_test_harness blocked, the module
imports cleanly and six tests FAIL by name carrying the full reason, while
the four source-inspection tests still pass.

KNOWN GAP, follow-up owed.  The four test_macos_rig_* cases assert on SOURCE
TEXT only, so behavioural mutants of tests/rig_vice_https_macos.py survive
them: reordering its gates at runtime, or changing a verdict without
changing the anchored text, would not be caught (mutants M5-M8 in review).
Closing that needs the rig restructured to be importable without the sibling
harness, which is not a drive-by.  Stated plainly: this commit pins the four
bridge rigs BEHAVIOURALLY and the macOS rig by SHAPE only.

SCOPE.  #178 stays split: Part 2 (the tools/test_pytest_boundary.py guard)
lands as #178b AFTER #172 and #177, because it codifies a rule those two
are instances of and pinning a rule while live violations remain is the
wrong order.  The consequence is explicit: this PR's guard will never be
SEEN to fail, and #178b owes the red-first demonstration.

A ninth instance of the class exists at tools/test_uci_data_acc.py:714-721
(missing PRG -> pytest.skip), in the exact lane require() was written for.
It is owned by open PR #172 and deliberately untouched here; a follow-up
is owed once that lands.  This PR's one-line pytest.ini addition was
3-way merge-tested against #172's pytest.ini: merges clean, no conflict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

REBASE NOTE (2026-09-05, onto dc06095).  Written at 0b55c30; #168, #172,
#173, #175 and #176 have landed since.  Re-verified, and one paragraph
above is now historical rather than current:

  - #172 is MERGED, so the "ninth instance" it owned is closed.  It is
    closed by a hand-rolled implementation of this same policy inside
    tools/test_uci_data_acc.py (`_require`/`Unavailable`, the "1"
    comparison, exit 2, the reason string carrying its own vacuity
    warning), not by importing require().  The follow-up that paragraph
    says is owed is therefore now actionable: converge that module onto
    tools/_skip_policy.py.  Not done here -- it is a separate change to a
    file this commit does not touch.
  - #177 is still OPEN, so the SCOPE paragraph stands: #178b still lands
    after it.
  - The only textual conflict on the rebase was pytest.ini's testpaths
    list, where #175 had added tools/test_runner_coverage.py.  Resolved
    by keeping all three, in alphabetical order.
  - Neither new module defines a module-level run_tests(), so #175's
    tools/test_runner_coverage.py guard does not claim them; it passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 85653f3399db790ac1a6c6e1057ce6cb0bac3505)

All four reproduce; each is fixed with a test that is red before and green
after.  Two of them are this PR committing the defect class it exists to
close, which is the reason to state them plainly rather than fold them in.

1. require()'s opt-out CRASHED outside pytest -- and the hand-rolled copy
   it supersedes had already fixed this.

   `import pytest` succeeds whenever pytest is INSTALLED, which on a
   developer machine is always; it says nothing about pytest DRIVING.  So
   `require()` called `pytest.skip()` in the standalone lane too, and
   `pytest.outcomes.Skipped` derives from BaseException, which neither new
   module's `except Exception` runner can catch.  Measured, with the
   sibling harness checkout blocked:

     C64_ALLOW_SKIP=1 python3 tools/test_rig_skip_contract.py
       -> Skipped raised out of require(), traceback, EXIT 1, no summary
          line, and the four source-inspection tests that could have run
          never ran.

   Exit 1 means "a check ran and failed" under this module's own contract,
   so it was wrong in both directions, and only on the escape hatch the PR
   advertises.  Without the opt-out the same run was correct (6 fail, 4
   pass, summary printed) -- which is why nothing caught it.

   tools/test_uci_data_acc.py:631 had already solved exactly this, with
   `sys.modules.get("pytest")` and a comment saying why: "Only hand the
   skip to pytest when pytest is actually driving; the standalone runner
   has its own reporting and must not see Skipped."  A shared helper that
   is WORSE than the copy it supersedes is not a refactor.  Fixed by
   mirroring it: outside pytest, require() now raises VoluntarySkip, a
   plain Exception the running lane can catch.  Either way the test body
   does not run, so the fail-closed property is unchanged.

   The old ImportError branch is gone with it: `sys.modules.get` answers
   "absent" and "not driving" the same way, and both want the same answer.

2. The literal-"1" guard was SKIP-SWALLOWED by the mutation it targets.

   `test_require_still_raises_when_the_opt_out_is_zero` was spelled
   `except SkipPolicyError: pass`, which is vacuous against its own
   defect: revert _opted_out() to bare truthiness and `bool("0")` is true,
   require() takes the opt-out branch, pytest records SKIPPED, exit 0.
   Measured under that mutant: `1 skipped`, not a failure.  The guard
   converted itself into a green skip under precisely the defect it
   guards -- the vacuous-skip shape #178 exists to kill, committed inside
   #178's own implementation, and it read as caught only because three
   sibling tests on the cannot_run() lane happen to fail.

   The catch is now BaseException-wide, because a Skipped here IS the bug.
   Under the same mutant the module now reports 4 failed, 13 passed.

   Two new cases pin defect 1 from both sides: the opt-out must raise
   VoluntarySkip when pytest is not driving, and must still be a pytest
   skip when it is -- so the fix cannot be "simplified" into never
   skipping, which would make the documented hatch a lie.

3. UNDISCLOSED LOOSENING: VICE_HTTPS_OK_TO_RUN unset went 2 -> 0.

   On master, tests/rig_phase3_https_1mhz.py printed "ABORT:
   VICE_HTTPS_OK_TO_RUN is not set." and returned 2, and its docstring
   documented `2 -- pre-flight gate refused`.  This PR replaced both with
   not_applicable() -> 0 and deleted that documented code, while the PR
   body disclosed only the opposite-direction tightening on --u64.

   The first draft's reasoning ("the operator declining an opt-in rig")
   does not survive contact with this repo's own taxonomy.  UNSET IS THE
   DEFAULT STATE: it cannot distinguish a considered decline from a
   forgotten flag, so exit 0 hands a green run to someone who verified
   nothing.  And what the flag asserts -- "the UCI HTTPS listener has
   stopped" -- is a CONTENTION claim, which is the one category this
   policy gives no opt-out at all; the port-443 check three lines below
   is the same interlock measured directly and was already exit 2.

   Reverted to exit 2, non-opt-out-able.  The only thing #178 changes
   about this gate now is that the verdict is a named block instead of a
   bare print.  Pinned behaviourally, including that C64_ALLOW_SKIP=1
   cannot rescue it.

4. tests/rig_phase2_http.py contradicted itself inside forty lines.

   Its module docstring said tests/rig_vice_https_macos.py "owns the
   coverage"; its own _COUNTERPART string -- the one that actually reaches
   the operator -- says that rig drives TLS, "so plaintext HTTP
   specifically has no macOS rig".  The docstring is what a reader hits
   first, and it is load-bearing here: the voluntary-skip verdict is
   justified by "another rig owns the coverage, nothing is lost", which
   for this one rig is false.  Coverage of plaintext HTTP over emulated
   RR-Net really is lost on macOS, at exit 0.

   Exit 0 stays -- there is no remedy on macOS and exit 2 would be a red
   nobody can clear -- but the reason is now the true one, in both the
   docstring and _wrong_platform(), and a new source-inspection test pins
   it so the two halves cannot drift apart again.

5. Minor, and the same root cause as 1: both _standalone() runners did
   `return 1 if failed else 0`, collapsing "could not run" onto 1 and
   contradicting the 0/1/2 contract of the module they test.  They now
   separate VoluntarySkip / SkipPolicyError / Exception and return
   0 / 1 / 2 accordingly.  Measured on tools/test_rig_skip_contract.py
   with the sibling harness blocked: exit 2 plain, exit 0 with
   C64_ALLOW_SKIP=1, exit 0 with the harness present.

Red-then-green, all four:

  D1  restore `import pytest` in require()
        -> test_require_does_not_hand_a_skip_to_a_non_pytest_runner FAILS
  D2  restore bare truthiness in _opted_out()
        -> test_require_still_raises_when_the_opt_out_is_zero FAILS
           (before this commit the same mutant made it SKIP, exit 0)
  D3  re-route the interlock to not_applicable()
        -> test_the_interlock_flag_unset_is_contention_and_stays_exit_two FAILS
  D4  restore the contradictory docstring
        -> test_phase2_does_not_claim_a_counterpart_it_does_not_have FAILS

Bare `pytest` at the repo root: 83 passed (was 80; +3 new cases).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Review round 2.  One real hole in a whole-PR claim, plus the fix from
round 1 that shipped without a test of its own.

F1.  "No exit code in this PR is weaker than master's" was FALSE by
     omission, at tools/test_ecdsa_p384_kat.py -- the one wired file the
     round-1 sweep did not touch, so it was never re-checked.

     Master, `--u64` with U64_HOST unset: prints a SKIP line inside the
     u64 branch, but the VICE lane has ALREADY RUN, and main() ends at
     `return 0 if total_fail == 0 else 1`.  A failing emulator vector
     therefore reported 1.

     This branch: the new gate fires before _build_prg() and before any
     VICE work, and carried opt_out_env="C64_ALLOW_SKIP".  With that set
     it returned 0 with nothing run at all.

     The defect is SCOPE, not arithmetic.  C64_ALLOW_SKIP answers "this
     lane has no hardware" -- which is precisely the operator who would
     set it here -- and honouring it at a gate that precedes the VICE
     lane silences the EMULATOR half too, which needs no hardware and
     could have run.  They get exit 0 and never learn the emulator-only
     half regressed.

     Fixed with opt_out_env=None rather than by moving the gate: the
     up-front refusal is the disclosed tightening (~0.1 s instead of
     hours), and moving it below the VICE lane would give that back.  The
     remedy needs no environment variable and costs nothing -- set
     U64_HOST, or drop --u64 and get the VICE lane on its own.  "This
     lane has no hardware" is spelled by NOT PASSING --u64; asking for
     hardware and not supplying it is a malformed invocation, not a
     coverage gap to acknowledge.

     The unreachable backstop later in main() gets the same treatment, so
     one file does not answer the same question two ways.

     Severity was low -- three deliberate operator choices, the block
     prints in full, and _build_prg() would likely fail first because the
     P-384 build has never completed -- but the claim was checkable and
     did not hold, so the code moved rather than the claim.

F3.  D5 (the _standalone() 0/1/2 split) was the one round-1 fix with no
     test.  Nothing collects either runner, so a future edit collapsing
     it back to `return 1 if failed else 0` would go unnoticed -- in a PR
     whose thesis is that a helper with no call-site test is just another
     convention to miss.

     Two subprocess cases in tools/test_skip_policy.py drive the REAL
     runner in tools/test_rig_skip_contract.py with c64_test_harness made
     unimportable by a meta_path finder (the sibling is pip-installed, so
     there is no path entry to remove).  Target chosen deliberately: it
     is the module with a genuine could-not-run lane, and driving it from
     here cannot recurse, because it has no spawner of its own.

       no opt-out       -> exit 2, "CANNOT RUN" and the vacuity string
                           both present, summary line still printed
       C64_ALLOW_SKIP=1 -> exit 0, no traceback, "skipped by explicit
                           opt-out", and the four tests that never needed
                           the sibling checkout still show PASS

     That second case is also the D1 crash scenario, now pinned: before
     the sys.modules fix this exact invocation died with an uncaught
     pytest Skipped at exit 1.

F2.  VoluntarySkip is an Exception, so a consumer's broad
     `except Exception` can swallow it and label it a failure.  Not
     reachable today -- every require() consumer names VoluntarySkip
     before the bare except -- and the direction is the safe one, since a
     swallowed voluntary skip reports as a FAILURE and never as a pass.
     One docstring paragraph says so, and tells a consumer to name it
     first.  Deliberately NOT a BaseException: that is the choice that
     caused D1.

Red-then-green:

  F1  restore opt_out_env="C64_ALLOW_SKIP" on the --u64 gate
        -> test_the_u64_gate_is_not_opt_out_able_because_it_precedes_the_vice_lane
           FAILS (1 failed, 11 passed)
  D5  collapse the runner to `return 1 if failed or cannot else 0`
        -> test_standalone_runner_returns_two_for_could_not_run
           FAILS (1 failed, 18 passed)

Bare `pytest` at the repo root: 86 passed (was 83; +3 new cases).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…te it

76621b6's comment on the new subprocess cases says the target module
"has no spawner of its own", so driving it from here cannot recurse.
True when I reasoned it; false by the time the same commit was written,
because that commit ALSO added a spawner to
tools/test_rig_skip_contract.py -- test_the_u64_gate_* runs
tools/test_ecdsa_p384_kat.py, and _standalone() runs every test_*.  So
the real chain is depth 2 and it fires:

  pytest
    -> test_rig_skip_contract.py   (spawned by test_skip_policy.py)
         -> test_ecdsa_p384_kat.py (spawned by its u64-gate test)

Measured rather than argued: inside the blocked standalone run the
u64-gate case reports PASS, which it can only do by having spawned the
KAT.  The nested run is 0.065 s end to end and exits 2, because the KAT's
gate returns before _build_prg() and before any VICE work.  Timeouts nest
correctly, 180 s outer and 120 s inner.

So: no fork bomb, no hang, nothing to fix in the code.  What needed
fixing is that the comment leaned on an invariant -- "no spawner exists"
-- which is exactly the property that would stop a third link being added
later, and which was no longer true.  A safety argument that has quietly
become false is worse than no argument, because the next person reads it
and stops looking.

The comment now spells the chain out, states the depth, says the bound is
by INSPECTION with no guard in the code, gives the measured cost and the
nesting of the timeouts, and records that the earlier claim went stale
inside one commit -- so a third link has to be added deliberately, in
sight of all that.

Comment-only.  Bare `pytest`: 86 passed, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

4e90ffc said the spawn chain "terminates because the P-384 KAT spawns
nothing".  Not true in general: _build_prg() runs `make clean` and two
`make BACKEND=uci` invocations at test_ecdsa_p384_kat.py:646-665.  The
next sentence already gave the real reason -- the --u64 gate returns
before _build_prg() -- so the load-bearing fact was present and the depth,
cost and timeout figures were all correct.  What was wrong is the
attribution, and it matters because of what it hides.

The KAT is a leaf ON THIS PATH ONLY, and the path is chosen by where that
gate sits.  So the cost bound of the D5 subprocess test depends on the F1
fix keeping the gate up front.  Move it below _build_prg() -- the
alternative fix considered and rejected when the gate was made
non-opt-out-able -- and the nested run starts a full `make clean && make`
inside a 120 s inner timeout.  Two decisions, one line apart in a file
that mentions neither, and nothing at the gate would tell you.

Both halves now say so:

  - tools/test_skip_policy.py's chain comment names the KAT's `make`
    calls with their line numbers, says the leaf property is a property
    of the PATH and not of the file, and states outright that moving the
    gate is also a decision about this test's cost.
  - tools/test_rig_skip_contract.py -- which ADDS the second link and
    carried no note at all, the comment living only in the module that
    adds the first -- gets the chain drawn at the spawn site, with the
    same warning and a pointer to the fuller note.

A cross-file invariant documented in exactly one of the two files is
half-documented: the reader who breaks it is the one editing the other.

Comment-only.  Bare `pytest`: 86 passed, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>